[TRT/TRT RTX EP] Fix bug for missing outputs in the returning ComputeCapability/IndexedSubGraph #26444

chilo-ms · 2025-10-29T23:52:04Z

Description

For TRT EP's GetCapability(), in some case, the GetSubGraph() won't add graph's output to the ComputeCapability/IndexedSubGraph returning to ORT.

The issue if from following code:

...
if (node->GetOutputEdgesCount() > node->OutputDefs().size()) {
 ... // execute here
} else {
  ...
          if (graph_output_names.find(output->Name()) != graph_output_names.end()) {
            graph_outputs_to_add[output] = output_order; // missing this
          }
}

Update TRT RTX EP as well.

Motivation and Context

#25373

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/test/providers/tensorrt/tensorrt_test_utils.cc

onnxruntime/test/providers/tensorrt/tensorrt_test_utils.h

onnxruntime/test/providers/tensorrt/tensorrt_test_utils.cc

onnxruntime/test/providers/tensorrt/tensorrt_test_utils.h

gcunhase · 2025-11-12T18:31:26Z

@chilo-ms any more AIs needed for this to be merged? Thanks.

yuslepukhin · 2025-11-19T22:55:33Z

onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc

-              graph_outputs_to_add[output] = output_order;
+              // This output is the graph's output.
+              // So the output should be put into the subgraph's output list.
+              graph_outputs_to_add.insert({output, output_order});


graph_outputs_to_add.insert({output, output_order});

It would not insert as before if the entry existed

If the key already exists in any of the those maps, i.e. fused_inputs,
fused_outputs, fused_outputs_to_add and graph_outputs_to_add, it's not necessarily to override it.

input_order/output_order is simply a relative ordering that associated with the input/output, so that when constructing final sub_graph's input of output lists from above maps, input/output with smaller order index will appear before those with larger order indices.

So exact order index is not necessary, as long as the order index for an output that should appear before another output has smaller order index is sufficient.

BTW, i added the comments for input_order/output_order to explain the usage of them.

yuslepukhin · 2025-11-19T22:57:31Z

onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc

-          if (graph.GetGraph().GetConsumerNodes(output->Name()).size() > 0) {
-            fused_outputs[output] = output_order++;
-          }
+    for (const auto& output : node->OutputDefs()) {


for (const auto& output : node->OutputDefs()) {

General comment for both inputs and outputs.
Should we check at some point if an optional input/output Exists()?

yuslepukhin · 2025-11-19T22:58:53Z

onnxruntime/test/providers/nv_tensorrt_rtx/nv_basic_test.cc

+   *
+   */
+  {
+    Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "test"};


Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "test"};

Just instantiate it once

yuslepukhin

🕐

onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc

edgchen1 · 2025-11-19T23:57:30Z

onnxruntime/test/providers/nv_tensorrt_rtx/nv_basic_test.cc

+   *                     |--- Mod ---> "labels"
+   */
+  {
+    Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "test"};


there's already a global Ort::Env instance here - why do we need to define another one? the underlying OrtEnv is a singleton so this would refer to the same instance.

changed to use global ort_env

edgchen1 · 2025-11-20T00:00:27Z

onnxruntime/test/providers/nv_tensorrt_rtx/test_nv_trt_rtx_ep_util.cc

+  return t;
+}
+
+OrtStatus* CreateModelWithNodeOutputNotUsed(const PathString& model_name) {


for other tests that use models, we have checked in .onnx files and a script to generate them. is there a good reason to do it differently here? defining the model in a Python script (e.g., with onnxscript) could be more concise.

okay, i added the python scripts as well as the models

onnxruntime/test/testdata/node_output_not_used.py

onnxruntime/test/testdata/topk_and_multiple_graph_outputs.py

onnxruntime/test/testdata/node_output_not_used.py

onnxruntime/test/testdata/topk_and_multiple_graph_outputs.py

onnxruntime/test/testdata/node_output_not_used.py

@@ -0,0 +1,43 @@
+import onnx


To resolve the "Module is imported with both 'import' and 'import from'" issue, remove the from onnx import TensorProto, helper statement and reference TensorProto and helper via the onnx module (that is, use onnx.TensorProto and onnx.helper). Update all usages of helper and TensorProto in the code accordingly. No additional dependencies or code structure changes are required. Only lines in onnxruntime/test/testdata/node_output_not_used.py handling imports and references to helper and TensorProto need to be changed.

onnxruntime/test/testdata/topk_and_multiple_graph_outputs.py

@@ -0,0 +1,78 @@
+import onnx


To address the issue, remove the from onnx import TensorProto, helper import, and instead refer to TensorProto and helper via the main namespace import: onnx.TensorProto and onnx.helper. This will make all references to ONNX symbols consistently qualified, improving code clarity. Specifically:

Remove line 2: from onnx import TensorProto, helper.

Change all references to TensorProto in this file to onnx.TensorProto.

Change all references to helper to onnx.helper.
No other functional changes are required. Only the one file onnxruntime/test/testdata/topk_and_multiple_graph_outputs.py needs editing.

update

1d92cb3

gcunhase mentioned this pull request Oct 30, 2025

[Bug] ORT inference outputs None value #25373

Open

Add unit tests

3f00973

chilo-ms marked this pull request as ready for review October 30, 2025 21:18

github-actions bot reviewed Oct 30, 2025

View reviewed changes

onnxruntime/test/providers/tensorrt/tensorrt_test_utils.cc Outdated Show resolved Hide resolved

onnxruntime/test/providers/tensorrt/tensorrt_test_utils.cc Outdated Show resolved Hide resolved

onnxruntime/test/providers/tensorrt/tensorrt_test_utils.h Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Oct 30, 2025

View reviewed changes

onnxruntime/test/providers/tensorrt/tensorrt_test_utils.cc Fixed Show fixed Hide fixed

onnxruntime/test/providers/tensorrt/tensorrt_test_utils.h Fixed Show fixed Hide fixed

chilo-ms added 3 commits October 30, 2025 14:30

lintrunner -a

4308033

update

fb44919

fix build issues

f763515

chilo-ms changed the title ~~[TensorRT EP] Fix bug for missing outputs in the returning ComputeCapability/IndexedSubGraph~~ [TRT/TRT RTX EP] Fix bug for missing outputs in the returning ComputeCapability/IndexedSubGraph Oct 31, 2025

fix build warning

186c938

chilo-ms requested review from adrianlizarraga, edgchen1 and yuslepukhin November 19, 2025 19:22

yuslepukhin reviewed Nov 19, 2025

View reviewed changes

yuslepukhin requested changes Nov 19, 2025

View reviewed changes

edgchen1 reviewed Nov 20, 2025

View reviewed changes

chilo-ms added 3 commits November 20, 2025 10:36

add comments and refactor the logic of adding graph's outputs

fe79531

fix typo

aef0602

address reviewer's comments

f169747

github-advanced-security bot found potential problems Nov 20, 2025

View reviewed changes

onnxruntime/test/testdata/node_output_not_used.py Fixed Show fixed Hide fixed

onnxruntime/test/testdata/topk_and_multiple_graph_outputs.py Fixed Show fixed Hide fixed

lintrunner -a

6c769c1

github-advanced-security bot found potential problems Nov 20, 2025

View reviewed changes

update

accf540

github-advanced-security bot found potential problems Nov 20, 2025

View reviewed changes

@@ -1,15 +1,14 @@
             import onnx
-            from onnx import TensorProto, helper
             def create_model_with_node_output_not_used(model_path):
                 # Create graph
-                X = helper.make_tensor_value_info("X", TensorProto.FLOAT, [3, 2])
-                W = helper.make_tensor_value_info("W", TensorProto.FLOAT, [2, 3])
-                Y = helper.make_tensor_value_info("Y", TensorProto.FLOAT, [2, 3])
+                X = onnx.helper.make_tensor_value_info("X", onnx.TensorProto.FLOAT, [3, 2])
+                W = onnx.helper.make_tensor_value_info("W", onnx.TensorProto.FLOAT, [2, 3])
+                Y = onnx.helper.make_tensor_value_info("Y", onnx.TensorProto.FLOAT, [2, 3])
                 # Dropout node (two outputs)
-                dropout_node = helper.make_node(
+                dropout_node = onnx.helper.make_node(
                     "Dropout",
                     inputs=["X"],
                     outputs=["dropout_out", "dropout_mask"],
@@ -17,21 +10,21 @@
                 )
                 # MatMul node
-                matmul_node = helper.make_node(
+                matmul_node = onnx.helper.make_node(
                     "MatMul",
                     inputs=["dropout_out", "W"],
                     outputs=["Y"],
                     name="MatMulNode",
                 )
-                graph = helper.make_graph(
+                graph = onnx.helper.make_graph(
                     nodes=[dropout_node, matmul_node],
                     name="DropoutMatMulGraph",
                     inputs=[X, W],
                     outputs=[Y],
                 )
-                model = helper.make_model(graph, opset_imports=[helper.make_operatorsetid("", 13)])
+                model = onnx.helper.make_model(graph, opset_imports=[onnx.helper.make_operatorsetid("", 13)])
                 onnx.checker.check_model(model)
                 onnx.save(model, model_path)

@@ -1,45 +1,44 @@
             import onnx
-            from onnx import TensorProto, helper
             def create_model_with_topk_graph_output(model_path):
                 # ======================
                 # ---- Inputs ----
                 # ======================
-                input_tensor = helper.make_tensor_value_info("input", TensorProto.FLOAT, ["N"])
+                input_tensor = onnx.helper.make_tensor_value_info("input", onnx.TensorProto.FLOAT, ["N"])
                 # ======================
                 # ---- Initializers ----
                 # ======================
-                K = helper.make_tensor("K", TensorProto.INT64, dims=[1], vals=[300])
-                zero = helper.make_tensor("zero", TensorProto.INT64, dims=[], vals=[0])
-                twenty_six = helper.make_tensor("twenty_six", TensorProto.INT64, dims=[], vals=[26])
+                K = onnx.helper.make_tensor("K", onnx.TensorProto.INT64, dims=[1], vals=[300])
+                zero = onnx.helper.make_tensor("zero", onnx.TensorProto.INT64, dims=[], vals=[0])
+                twenty_six = onnx.helper.make_tensor("twenty_six", onnx.TensorProto.INT64, dims=[], vals=[26])
                 # ======================
                 # ---- Nodes ----
                 # ======================
-                topk_node = helper.make_node(
+                topk_node = onnx.helper.make_node(
                     "TopK",
                     inputs=["input", "K"],
                     outputs=["scores", "topk_indices"],
                     name="TopK",
                 )
-                less_node = helper.make_node(
+                less_node = onnx.helper.make_node(
                     "Less",
                     inputs=["topk_indices", "zero"],
                     outputs=["Less_output_0"],
                     name="Less",
                 )
-                div_node = helper.make_node(
+                div_node = onnx.helper.make_node(
                     "Div",
                     inputs=["topk_indices", "twenty_six"],
                     outputs=["Div_17_output_0"],
                     name="Div",
                 )
-                mod_node = helper.make_node(
+                mod_node = onnx.helper.make_node(
                     "Mod",
                     inputs=["topk_indices", "twenty_six"],
                     outputs=["labels"],
@@ -49,15 +16,15 @@
                 # =========================
                 # ---- Graph Outputs ----
                 # =========================
-                scores_out = helper.make_tensor_value_info("scores", TensorProto.FLOAT, ["K"])
-                less_out = helper.make_tensor_value_info("Less_output_0", TensorProto.BOOL, ["K"])
-                div_out = helper.make_tensor_value_info("Div_17_output_0", TensorProto.INT64, ["K"])
-                labels_out = helper.make_tensor_value_info("labels", TensorProto.INT64, ["K"])
+                scores_out = onnx.helper.make_tensor_value_info("scores", onnx.TensorProto.FLOAT, ["K"])
+                less_out = onnx.helper.make_tensor_value_info("Less_output_0", onnx.TensorProto.BOOL, ["K"])
+                div_out = onnx.helper.make_tensor_value_info("Div_17_output_0", onnx.TensorProto.INT64, ["K"])
+                labels_out = onnx.helper.make_tensor_value_info("labels", onnx.TensorProto.INT64, ["K"])
                 # ======================
                 # ---- Graph ----
                 # ======================
-                graph = helper.make_graph(
+                graph = onnx.helper.make_graph(
                     nodes=[topk_node, less_node, div_node, mod_node],
                     name="TopKGraph",
                     inputs=[input_tensor],
@@ -65,7 +28,7 @@
                     initializer=[K, zero, twenty_six],
                 )
-                model = helper.make_model(graph, opset_imports=[helper.make_operatorsetid("", 13)])
+                model = onnx.helper.make_model(graph, opset_imports=[onnx.helper.make_operatorsetid("", 13)])
                 # Validate + Save
                 onnx.checker.check_model(model)

[TRT/TRT RTX EP] Fix bug for missing outputs in the returning ComputeCapability/IndexedSubGraph #26444

Are you sure you want to change the base?

[TRT/TRT RTX EP] Fix bug for missing outputs in the returning ComputeCapability/IndexedSubGraph #26444

Conversation

chilo-ms commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gcunhase commented Nov 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chilo-ms Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check notice

Copilot Autofix

Check notice

Copilot Autofix

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chilo-ms commented Oct 29, 2025 •

edited

Loading

chilo-ms Nov 20, 2025 •

edited

Loading